Skip to content

feat(ai,ai-fal): per-model typed durations for video generation#641

Draft
tombeckenham wants to merge 19 commits into
619-create-schemas-libraryfrom
534-video-duration-support
Draft

feat(ai,ai-fal): per-model typed durations for video generation#641
tombeckenham wants to merge 19 commits into
619-create-schemas-libraryfrom
534-video-duration-support

Conversation

@tombeckenham
Copy link
Copy Markdown
Contributor

Closes #534. Built on top of #622 (@tanstack/ai-schemas) — targets that branch as the base. When #622 merges into main, this PR will rebase + retarget to main.

Summary

  • New fifth generic on VideoAdapter (TModelDurationByName) + two introspection methods on the base class — availableDurations() and snapDuration(seconds). Default implementations return { kind: 'none' } / undefined, so video adapters that haven't been migrated keep working unchanged.
  • generateVideo({ duration }) is now per-model typed via VideoDurationForAdapter<TAdapter>.
  • FAL adapter derives duration per model from @fal-ai/client's EndpointTypeMap. Runtime constraint data for popular models lives in a hand-curated map in packages/typescript/ai-fal/src/video/video-provider-options.ts (to be replaced with schema-derived lookups once feat(schemas): @tanstack/ai-schemas with nightly OpenAPI sync (closes #619) #622's FAL pipeline syncs).
  • New shared snapToDurationOption util handles discrete enums, ranges, mixed, keyword-with-unit forms ('8s'), and kind: 'none'.

Per-model behaviour after this PR

Model duration type availableDurations()
fal-ai/kling-video/v1.6/{standard,pro}/text-to-video '5' | '10' discrete
fal-ai/pika/v2.2/text-to-video '5' | '10' discrete
fal-ai/luma-dream-machine/ray-2 '5s' | '9s' discrete
fal-ai/veo3 and fal-ai/veo3/image-to-video '4s' | '6s' | '8s' discrete
fal-ai/wan-25-preview/text-to-video '2' … '15' discrete
fal-ai/minimax/video-01 not accepted { kind: 'none' }
fal-ai/hunyuan-video-v1.5/text-to-video not accepted (uses num_frames) { kind: 'none' }

Scope decisions

  • Sora out of scope. OpenAI's video adapter was reverted to its pre-PR state at the user's request. It picks up the base-class defaults ({ kind: 'none' }) until someone migrates it.
  • Gemini Veo follow-up. There is no Veo adapter today (model-meta entries commented out at packages/typescript/ai-gemini/src/model-meta.ts:827-940). Filed as feat(gemini): add Google Veo video adapter on the typed-duration contract #634 to build directly on this contract.

Breaking change

Callers passing duration: <number> to FAL video models must either:

  • pass the typed string union ('5', '8s', etc.) directly, or
  • call adapter.snapDuration(seconds) and let it coerce.

Existing video adapters that haven't opted into the typed-duration map (OpenAI Sora today) are not breaking.

Test plan

  • pnpm test:types — repo-wide type check passes
  • pnpm test:lib — all 32 packages, 904 core tests + 86 FAL tests pass (5 new snap.test.ts cases + 4 new FAL durations cases)
  • pnpm test:eslint — clean
  • pnpm build — all 32 projects build clean
  • Recording a FAL fixture for E2E once FAL_KEY is available; the E2E suite's video-gen matrix currently doesn't include 'fal' so this is also follow-up infra work.
  • Manual TS check in an editor: falVideo('fal-ai/veo3') autocompletes '4s' | '6s' | '8s' on duration; falVideo('fal-ai/minimax/video-01') rejects duration.

🤖 Generated with Claude Code

tombeckenham and others added 18 commits May 22, 2026 15:10
…oses #619)

Adds a separate `@tanstack/ai-schemas` package that ships per-endpoint
JSON Schema + Zod definitions for every supported provider, generated
nightly from upstream OpenAPI specs.

Architecture ported from fal-ai/fal-js PR #212 and generalised to
multi-provider. Pipeline: `fetch-schemas` (per-provider fetchers) →
`generate-schemas` (`@hey-api/openapi-ts` with the Zod 4 plugin) →
`generate-endpoint-maps` (bundles `$ref` closures under `$defs`, emits
endpoint-id-keyed maps and namespaced top-level barrels).

Providers wired up with working generation:
- OpenAI: `github.com/openai/openai-openapi` raw YAML
- Anthropic: Stainless OpenAPI resolved via anthropic-sdk-typescript/.stats.yml
- Gemini: Google Discovery doc → OpenAPI converted in-pipeline
- ElevenLabs: `api.elevenlabs.io/openapi.json`
- FAL: per-model OpenAPI from `api.fal.ai/v1/models` (skips when FAL_KEY absent)

.github/workflows/sync-schemas.yml runs daily and opens an automated PR
when any upstream spec diff lands, following the same pattern as
sync-models.yml.

Per repo coupling rules: media-generation + adapter-configuration
SKILL.md cross-reference the new package; new docs/advanced/ai-schemas.md
page added under the Advanced section.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Switches `@tanstack/ai-schemas` from format-first
(`@tanstack/ai-schemas/schemas/openai`, `/zod/gemini`) to provider-first
(`@tanstack/ai-schemas/openai/json-schema`, `/gemini/zod`) and drops the
namespaced aggregator barrels.

With provider-first subpaths and no aggregator, importing one provider's
schemas pulls only that provider's bytes. Importing
`@tanstack/ai-schemas/gemini/json-schema` carries no OpenAI, Anthropic,
ElevenLabs, or FAL code into the consumer's bundle.

The default `.` entry now only re-exports `toOpenAIStrict` (4 KB).

Wildcard exports in package.json:

  "./*/json-schema" -> dist/esm/providers/*/schemas-index.{js,d.ts}
  "./*/zod"         -> dist/esm/providers/*/index.{js,d.ts}

Generator emits a flat layout — `providers/{providerId}` for single-
category providers, `providers/{providerId}-{category}` for FAL
(`fal-image`, `fal-video`, etc.) — so Node's single-segment `*` wildcard
resolves cleanly.

Vite discovers per-provider barrels at build time so the dist mirrors
the provider layout.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds Grok as a provider in the nightly OpenAPI sync. Source is xAI's
first-party OpenAPI spec at docs.x.ai/openapi.json (OpenAPI 3.1.0, 32
endpoints, 168 schemas). Public, no auth required for the spec.

Grok is API-compatible with OpenAI's chat completions at the wire level
but ships an independent schema set (different names, slightly different
shapes, plus xAI-specific endpoints: deferred-completion, documents/search,
per-modality model lists, video edits/extensions, tokenize-text, etc.).
Each provider stays isolated in the schemas package.

Bumps the build script's heap to 8 GB. With six providers now in the
single dts program, the default 4 GB limit was tight. Knip ignores
`vite` for ai-schemas since the script invokes the binary directly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…bles

The earlier `/* eslint-disable */` and `// @ts-nocheck` headers on every
generated file were holdovers from iterations before the spec-level
transforms (`coerceDefaults`, discriminator rewrites) cleaned up the
codegen. With those in place, the only real issues are:

- One `no-control-regex` lint error in `elevenlabs/zod.gen.ts` (a literal
  `\x09` in a regex pattern, faithful to upstream).
- One tsc error in `openai/zod.gen.ts` where the post-process rewrites
  `.extend()` on a discriminatedUnion to `z.intersection(...)` (runtime
  correct, but intersections aren't `$ZodTypeDiscriminable`).

Replace the blanket suppressors with two surgical mechanisms:

- `suppressControlRegexLines` injects
  `// eslint-disable-next-line no-control-regex` immediately above each
  regex literal containing a control-char escape.
- The `.extend()` rewriter now reports whether it touched anything; only
  files where it did get a two-line header (`ban-ts-comment` disable
  followed by `@ts-nocheck`) so the workspace lint rule doesn't fire.

Net effect: zero file-level disables, two line-level disables on the only
file that needs them, ~100 `no-explicit-any` warnings (warn-level in the
project config, do not fail CI). `pnpm test:eslint` and `pnpm test:types`
both walk the generated files in full and pass clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…rkflow

`@hey-api/openapi-ts` runs with full code-execution privileges during
nightly codegen, so an unintentional bump shouldn't be possible. Drop the
caret on the version specifier (`^0.97.2` -> `0.97.2`) so future updates
have to land via an explicit PR. The lockfile already pinned the resolved
version; this just removes the wiggle room in the manifest.

Add a `pnpm audit --audit-level=high` step to sync-schemas.yml. It's
non-blocking (`continue-on-error: true`) because the workspace currently
carries pre-existing transitive advisories (e.g. an old protobufjs via
@google/genai used by ai-gemini) that aren't related to schemas codegen
and shouldn't be allowed to break the nightly. The step's purpose is to
surface any *new* advisory landing on the codegen dep tree so the human
reviewing the automated PR can spot it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
We already have pnpm's trust-downgrade policy active (refuses installs
where provenance attestation regressed), the heyapi version explicitly
pinned without a caret, and an automated PR pattern that forces human
review before any codegen change lands. A non-blocking audit step would
just be log noise, and a blocking one needs scope work we don't want to
take on right now.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* feat(ai-openai): add gpt-image-2 to image model meta

Adds `gpt-image-2` to OPENAI_IMAGE_MODELS so it can be used through
openaiImage adapters. Reuses the gpt-image-1 provider-options/size
shape (quality, background, output_format, output_compression,
moderation, partial_images; sizes 1024x1024 / 1536x1024 / 1024x1536
/ auto) and extends size + prompt-length validators. Also updates
the media-generation skill and image-generation doc page to list
the new model.

* fix(ai-openrouter): restore web_fetch in tool capabilities map

The model-metadata sync in #623 regenerated
`OpenRouterChatModelToolCapabilitiesByName` with `['web_search']` only,
which made `webFetchTool()` (added in #611) unassignable to any
OpenRouter text adapter and broke the per-model type-safety test.

Add `'web_fetch'` back so the existing tests compile.
* docs: refresh README discoverability

* docs: address README review feedback
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix: complete tool calls with server results

* fix: hydrate server tool outputs from history

* test: cover server tool history hydration

* ci: apply automated fixes

* test: add issue 176 manual repro page

* ci: apply automated fixes

* test: add live issue 176 repro flow

* test: hide issue 176 repro from sidebar

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Adds a fifth generic `TModelDurationByName` to `VideoAdapter` plus two
introspection methods on the base class:

- `availableDurations()` returns a `DurationOptions` tagged union
  (`discrete | range | mixed | none`) describing the durations the
  current model accepts.
- `snapDuration(seconds)` coerces raw seconds to the closest valid
  duration for the current model.

`generateVideo({ duration })` is now typed via
`VideoDurationForAdapter<TAdapter>`. The FAL adapter derives its
per-model duration type from the SDK's `EndpointTypeMap`, so e.g.
`falVideo('fal-ai/kling-video/v1.6/standard/text-to-video')` types
`duration` as `'5' | '10'`; `falVideo('fal-ai/veo3')` types it as
`'4s' | '6s' | '8s'`; `falVideo('fal-ai/minimax/video-01')` rejects
the field entirely.

Adapters that have not yet declared their per-model duration map get
sensible defaults (`{ kind: 'none' }`, `undefined`) so existing video
adapters keep working without changes.

Built on top of #622 (`@tanstack/ai-schemas`); once that PR's FAL
pipeline syncs runtime constraint data, the hand-curated map in
`packages/typescript/ai-fal/src/video/video-provider-options.ts` can
be replaced with schema-derived lookups. Follow-up issue #634 covers
building the Gemini Veo adapter directly on this contract.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@tombeckenham tombeckenham requested a review from a team as a code owner May 26, 2026 04:23
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 26, 2026

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 8c953f31-e7ae-4852-b673-85a09604e7f0

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch 534-video-duration-support

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 26, 2026

🚀 Changeset Version Preview

3 package(s) bumped directly, 28 bumped as dependents.

🟥 Major bumps

Package Version Reason
@tanstack/ai 0.21.3 → 1.0.0 Changeset
@tanstack/ai-fal 0.7.13 → 1.0.0 Changeset
@tanstack/ai-anthropic 0.10.3 → 1.0.0 Dependent
@tanstack/ai-code-mode 0.1.20 → 1.0.0 Dependent
@tanstack/ai-code-mode-skills 0.1.20 → 1.0.0 Dependent
@tanstack/ai-elevenlabs 0.2.10 → 1.0.0 Dependent
@tanstack/ai-event-client 0.3.10 → 1.0.0 Dependent
@tanstack/ai-gemini 0.10.10 → 1.0.0 Dependent
@tanstack/ai-grok 0.8.7 → 1.0.0 Dependent
@tanstack/ai-groq 0.2.6 → 1.0.0 Dependent
@tanstack/ai-isolate-node 0.1.20 → 1.0.0 Dependent
@tanstack/ai-isolate-quickjs 0.1.20 → 1.0.0 Dependent
@tanstack/ai-ollama 0.6.21 → 1.0.0 Dependent
@tanstack/ai-openai 0.10.0 → 1.0.0 Dependent
@tanstack/ai-openrouter 0.9.7 → 1.0.0 Dependent
@tanstack/ai-preact 0.6.32 → 1.0.0 Dependent
@tanstack/ai-react 0.11.7 → 1.0.0 Dependent
@tanstack/ai-react-ui 0.8.1 → 1.0.0 Dependent
@tanstack/ai-solid 0.10.7 → 1.0.0 Dependent
@tanstack/ai-solid-ui 0.7.1 → 1.0.0 Dependent
@tanstack/ai-svelte 0.10.7 → 1.0.0 Dependent
@tanstack/ai-vue 0.10.8 → 1.0.0 Dependent
@tanstack/openai-base 0.3.6 → 1.0.0 Dependent

🟨 Minor bumps

Package Version Reason
@tanstack/ai-schemas 0.1.0 → 0.2.0 Changeset

🟩 Patch bumps

Package Version Reason
@tanstack/ai-client 0.11.7 → 0.11.8 Dependent
@tanstack/ai-devtools-core 0.3.37 → 0.3.38 Dependent
@tanstack/ai-isolate-cloudflare 0.2.11 → 0.2.12 Dependent
@tanstack/ai-vue-ui 0.2.3 → 0.2.4 Dependent
@tanstack/preact-ai-devtools 0.1.41 → 0.1.42 Dependent
@tanstack/react-ai-devtools 0.2.41 → 0.2.42 Dependent
@tanstack/solid-ai-devtools 0.2.41 → 0.2.42 Dependent

@nx-cloud
Copy link
Copy Markdown

nx-cloud Bot commented May 26, 2026

View your CI Pipeline Execution ↗ for commit a4ed1b6


☁️ Nx Cloud last updated this comment at 2026-05-26 04:25:19 UTC

@nx-cloud
Copy link
Copy Markdown

nx-cloud Bot commented May 26, 2026

View your CI Pipeline Execution ↗ for commit a4ed1b6

Command Status Duration Result
nx run-many --targets=build --exclude=examples/... ✅ Succeeded 1m 40s View ↗

☁️ Nx Cloud last updated this comment at 2026-05-26 04:27:16 UTC

@pkg-pr-new
Copy link
Copy Markdown

pkg-pr-new Bot commented May 26, 2026

Open in StackBlitz

@tanstack/ai

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai@641

@tanstack/ai-anthropic

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-anthropic@641

@tanstack/ai-client

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-client@641

@tanstack/ai-code-mode

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-code-mode@641

@tanstack/ai-code-mode-skills

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-code-mode-skills@641

@tanstack/ai-devtools-core

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-devtools-core@641

@tanstack/ai-elevenlabs

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-elevenlabs@641

@tanstack/ai-event-client

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-event-client@641

@tanstack/ai-fal

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-fal@641

@tanstack/ai-gemini

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-gemini@641

@tanstack/ai-grok

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-grok@641

@tanstack/ai-groq

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-groq@641

@tanstack/ai-isolate-cloudflare

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-isolate-cloudflare@641

@tanstack/ai-isolate-node

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-isolate-node@641

@tanstack/ai-isolate-quickjs

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-isolate-quickjs@641

@tanstack/ai-ollama

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-ollama@641

@tanstack/ai-openai

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-openai@641

@tanstack/ai-openrouter

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-openrouter@641

@tanstack/ai-preact

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-preact@641

@tanstack/ai-react

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-react@641

@tanstack/ai-react-ui

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-react-ui@641

@tanstack/ai-schemas

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-schemas@641

@tanstack/ai-solid

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-solid@641

@tanstack/ai-solid-ui

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-solid-ui@641

@tanstack/ai-svelte

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-svelte@641

@tanstack/ai-utils

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-utils@641

@tanstack/ai-vue

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-vue@641

@tanstack/ai-vue-ui

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-vue-ui@641

@tanstack/openai-base

npm i https://pkg.pr.new/TanStack/ai/@tanstack/openai-base@641

@tanstack/preact-ai-devtools

npm i https://pkg.pr.new/TanStack/ai/@tanstack/preact-ai-devtools@641

@tanstack/react-ai-devtools

npm i https://pkg.pr.new/TanStack/ai/@tanstack/react-ai-devtools@641

@tanstack/solid-ai-devtools

npm i https://pkg.pr.new/TanStack/ai/@tanstack/solid-ai-devtools@641

commit: a4ed1b6

@tombeckenham tombeckenham marked this pull request as draft May 26, 2026 04:58
@tombeckenham tombeckenham force-pushed the 619-create-schemas-library branch from 7a05fca to d9904cf Compare June 5, 2026 08:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants